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Introduction 

The  TREC-6  Interactive  Track  concentrated  on  Aspect-Oriented  Retrieval:  finding  at  least  one 
document  that  covers  each  aspect  of  relevance  to  a  given  topic,  as  opposed  to  the  usual 
information-retrieval  task  of  simply  finding  documents  containing  as  much  information  as 
possible  about  the  topic.  For  this  track,  we  at  the  Center  for  Intelligent  Information  Retrieval 
of  the  University  of  Massachusetts  designed  and  ran  our  first-ever  user  study. 


Hypotheses 

We  designed  our  systems  with  several  hypotheses  in  mind  about  how  best  a  system  can  help  a 
user  with  aspect  retrieval.  The  hypotheses  that  we  were  able  to  test  in  this  experiment  were: 

1.  A  tool  that  is  designed  specifically  to  help  the  searcher  keep  track  of  aspects  will  be 
helpful. 

2.  A  searcher  will  be  likely  to  submit  several  queries  on  the  same  topic,  so  will  benefit 
from  a  means  for  indicating  which  documents  were  seen  on  earlier  queries. 

3.  Extracting  the  few  most  significant  terms  from  a  set  of  grouped  documents  (that 
represent  an  aspect)  will  help  the  searcher  meaningfully  label  that  aspect. 

4.  Ability  to  use  a  3-D  visualization  will  correlate  with  testable  structural  visualization 
ability. 


Experimental  Design 

The  TREC-6  interactive  track  experimental  design  specification  [9]  calls  for  six  topics 
especially  chosen  from  the  TREC-6  adhoc  topics  and  modified  for  use  in  the  interactive 
track.  The  corpus  consists  of  newspaper  articles  from  the  Financial  Times  of  London  1991- 
1994,  approximately  200,000  articles  in  all.  This  corpus  is  a  subset  of  the  TREC 
collection[5].  The  basic  experimental  design  calls  for  four  participants,  each  running  all  six 
topics,  but  three  with  the  control  system  and  three  with  the  experimental  system.  The  order  of 
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topics  is  held  the  same  for  all  participants  but  the  systems  are  alternated:  half  use  the  control 
system  first,  and  half  use  the  experimental  system  first. 

This  basic  design  leads  to  six  2x2  Latin  squares  to  estimate  the  difference  between  the  control 
and  experimental  systems.  We  augmented  the  design  as  suggested  by  the  specification, 
namely  by  adding  more  participants  in  groups  of  four  to  replicate  the  4x6  design,  and  by 
adding  a  second  experimental  system  and  using  the  same  4x6  design  for  it.  The  basic  design 
and  these  augmentations  account  for  our  groups  1  through  4.  In  addition,  we  had  a  fifth 
group  use  both  experimental  systems  and  no  control  system;  this  was  not  part  of  the 
experiment  proper,  and  this  group’s  searches  will  not  affect  results  tabulated  by  NIST— 
though  documents  they  chose  will  be  judged  by  the  NIST  assessors.  (See  Table  1.) 


Experimental  Systems 

As  required  by  NIST,  we  ran  their  ZPRISE  as  a  control  system;  the  two  experimental  systems 
were  basic  and  extended  versions  of  one  program.  The  extended  version  (“Asplnquery 
Plus”)  simply  added  a  3-D  window  to  the  basic  system  (“Asplnquery”)  .  Both  versions  use 
the  well-known  Inquery  search  engine.  The  core  of  our  user  interface  has  much  in  common 
with  the  ZPRISE  interface,  differing  in  two  significant  ways:  ZPRISE  displays  the  query  terms 
contained  in  a  document  after  the  headline  but  our  system  does  not,  and  our  system  color 
codes  whether  a  document  has  been  viewed  but  ZPRISE  does  not.  Specifically,  we  write  the 
headline  information  of  a  document  in  blue  if  it  has  not  been  viewed  before,  and  purple  if  it 
has  been  seen.  (This  scheme  was  modeled  after  the  default  color  scheme  Web  browsers  use  to 
show  if  a  hypertext  link  has  been  followed  or  not.) 

Both  ZPRISE  and  our  system  accept  plain  text  input  for  queries.  Our  system  also  supports  a 
phrase  operator,  invoked  by  placing  terms  together  within  double  quotes  (e.g.,  "balanced 
budget").  The  phrase  operator  increases  the  ranking  assigned  to  documents  where  all  terms  in 
the  phrase  are  found  in  close  proximity.  (In  reality,  our  system  supports  the  full  syntax  of 
Inquery,  dozens  of  operators  in  all,  but  this  is  the  only  one  we  told  participants  about.) 

Aspect  Window 

With  a  basic  IR  system,  an  analyst  may  be  able  to  find  the  documents  containing  various 
aspects,  but  he  or  she  has  to  use  another  window  or  a  piece  of  paper  to  keep  track  of  what  has 
been  found  already.  We  implemented  an  “aspect  window”  tool  to  help  with  this  task.  The 
idea  is  to  provide  an  area  where  documents  on  a  particular  aspect  can  be  stored.  To  help  label 
the  information,  statistical  analysis  of  word  and  phrase  occurrences  is  used  to  decide  what 
terms  and  phrases  are  most  distinctive  about  a  document  or  set  of  documents  in  an  aspect.  We 
provided  an  area  for  the  user  to  manually  assign  additional  keywords  or  labels  if  needed. 

Each  area  of  the  aspect  window  has  a  colored  border,  a  text  field  at  the  top  for  entering  a 
descriptive  label,  and  an  automatically  generated  list  of  the  five  noun  phrases  that  most 
distinguish  the  group  of  documents  assigned  to  this  aspect  from  the  remainder  of  the 
collection.  The  description  field  is  solely  for  the  user’s  convenience  and  need  not  be  filled.  If 
the  user  wants  a  description  they  can  type  or  paste  into  it,  or  drag  automatically  generated 
phrases  into  it.  Figure  1  shows  an  example  of  the  aspect  window.  The  system  shows  two 
groups  of  documents  (two  aspects)  already  identified  and  a  third  area  waiting  for  the  next 
aspect.  The  first  aspect  contains  one  document,  91512.  The  user  entered  this  document  into 
the  aspect  by  dragging  a  listed  document  from  the  ranked  list  display  (part  of  the  basic 
interface)  into  the  aspect's  document  list.  The  system  then  analyzed  the  selected  document 
and  found  five  phrases  that  describe  the  aspect:  the  four  shown  are  “alzheimer”,  “app”, 
“dementia”,  “brain”,  and  the  phrase  “brain  cell”.  The  analyst  did  not  find  those  phrases 
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descriptive  enough,  so  he  or  she  manually  inserted  the  term  “velnacrine”.  For  the  second 
aspect  the  automatically  identified  term  “sumatriptan”  is  an  adequate  descriptor. 

The  purpose  of  the  aspect  window  is  to  assist  the  user  in  categorizing  the  information  as  it  is 
discovered,  and  to  keep  an  overview  of  the  information  discovered  so  far.  In  an  aspect- 
oriented  or  briefing  type  setting  this  step  is  required  for  the  task  to  be  completed  properly, 
but  to  our  knowledge  no  systems  have  been  built  so  far  which  provide  any  assistance  for  this 
task. 

Visualization:  3-D  Window 

Another  important  step  in  the  aspect  oriented  retrieval  task  is  deciding  (repeatedly)  which 
document  to  look  at  next.  In  a  ranked  retrieval  system  the  documents  are  presented  in  the 
order  of  probability  of  relevance,  so  the  user  is  more  likely  to  encounter  relevant  documents 
at  the  top  of  the  list  than  further  down.  Nearly  always  the  headline  is  used  to  decide  if  the  full 
text  is  worth  reviewing  or  not.  Some  systems  [6,  11],  ZPRISE  among  them,  give  information 
about  the  query  terms  that  appear  in  the  document,  expecting  that  they  can  be  used  to  help 
decide  whether  to  investigate  further.  But  for  an  aspect  retrieval  task,  the  deciding  point  of 
whether  to  investigate  a  document  further  is  not  the  information  content,  but  the  marginal 
information  content-i.e.,  the  information  content  in  the  context  of  what  has  already  been 
seen.  We  believe  that  documents  that  are  similar  in  terms  of  information  content  will  also  be 
lexically  similar.  The  Cluster  Hypothesis  [10]  states  that  relevant  documents  tend  to  cluster, 
and  it  has  been  shown  to  be  valid  in  top-ranked  documents  [3,  7],  Aspects  represent  different 
forms  of  relevance,  and  we  believe  that  they  will  group  together  within  the  set  of  relevant 
documents. 

Asplnquery  Plus  compares  documents  in  an  extremely  high-dimensional  space 
(approximately  400,000  for  this  collection)  where  each  dimension  corresponds  to  a  feature  in 
the  collection  and  the  distance  was  measured  by  the  sine  of  the  angle  between  the  vectors. 
That  space  was  collapsed  to  three  dimensions  for  visualization  using  a  spring  embedding 
algorithm  [8].  The  interface  included  a  slider  for  adjusting  the  threshold  that  determined  the 
tightness  of  the  generated  clusters.  The  resulting  visualization  is  similar  in  style  to  BEAD  [2], 
differing  in  a  few  key  aspects:  BEAD  was  used  on  an  entire  (though  small)  corpus,  and  this 
display  is  used  only  on  the  retrieved  set;  the  vectors  used  by  BEAD  were  based  on  document 
abstracts  but  the  vectors  used  by  this  display  are  based  on  full  text. 

Documents  that  are  nearby  in  3-space  are  generally  nearby  in  the  high  dimensional  space 
also  (though  the  spring  embedding  dimensional  reduction  occasionally  forces  unrelated 
documents  to  be  near  one  another),  meaning  that  they  share  information  content  to  a 
considerable  degree.  For  that  reason,  the  3-D  display  provides  the  user  with  information 
about  whether  the  document  is  worth  investigating  further,  helping  the  user  to  sort  through 
documents  more  quickly.  Documents  that  have  not  been  assigned  to  any  aspect  have  the  same 
blue/purple  (read/unread)  color  scheme  that  is  used  in  the  main  window.  Documents  in  the  3- 
D  window  are  persistent  between  queries:  when  new  documents  are  retrieved  they  are  colored 
light  blue  (light  purple  when  read)  and  are  placed  in  the  3-D  window  by  the  forces  exerted 
from  already  placed  documents.  Figure  2  shows  five  newly  retrieved  documents  in  light  gray. 
It  is  easy  to  see  that  three  of  these  documents  fall  into  a  group  of  two  previously  seen 
documents  (upper  right  of  figure)  and  the  other  new  documents  fall  into  the  small  group  in 
the  upper  left  and  the  large  group.  An  analyst  who  is  under  time  pressure  could  use  the  3-D 
display  to  decide  that  the  unjudged  document  near  that  aspect  is  probably  on  the  same  aspect 
and  so  not  worth  examining.  A  retrieved  document  that  is  far  from  any  already-marked 
aspect  is  more  likely  to  be  useful.  (We  have  been  investigating  variations  on  the  visualization 
that  enhance  the  ability  for  a  user  to  find  new  and  interesting  material.  [1]) 
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The  three  windows— result  list,  aspect,  and  3-D— were  tightly  integrated.  If  a  document  is 
selected  by  a  mouse  click  in  any  of  the  three  windows,  that  document  is  highlighted  in  all 
windows  in  which  it  is  visible.  A  document  can  be  opened  for  viewing  by  double  clicking  in 
any  of  the  three  windows.  The  colors  were  coordinated  between  the  windows:  if  a  document 
has  been  saved  to  an  aspect,  that  aspect's  color  is  assigned  to  the  document  in  the  3-D  window 
and  also  placed  before  the  document  in  the  list. 


Participants 

We  were  interested  in  how  librarians  perform  search  tasks  as  compared  to  a  more  general  user 
population.  To  that  end,  we  recruited  20  participants:  eight  librarians  and  12  general  users. 
Table  1  shows  the  types  of  participants  in  and  the  systems  used  by  the  different  groups  in  the 
experiment. 

The  librarians  were  enlisted  via  personal  contacts.  All  were  staff  members  at  the  University  of 
Massachusetts/Amherst;  all  had  MLS  degrees,  and  several  had  an  additional  graduate  degree. 
Seven  of  the  eight  librarians  were  over  forty  (the  eighth  was  in  their  twenties).  Six  of  the 
librarians  were  women  and  two  were  men.  The  librarians  had  very  substantial  experience  with 
online  searching,  over  periods  ranging  from  2  to  20  years  and  averaging  10.  Furthermore, 
they  had  considerable  experience  in  every  category  of  searching  we  asked  about  (library 
catalogs,  CDROMs,  commercial  services,  and  the  WWW).  They  had  relatively  little  experience 
with  ranked  lists  or  relevance  feedback. 

Of  the  general  participants,  10  were  recruited  by  flyers  distributed  on  campus,  while  the  other 
two  were  neighbors  of  a  staff  member.  (We  actually  had  over  20  responses  to  the  flyers;  we 
accepted  volunteers  on  a  first  come/first  served  basis.)  In  most  ways  this  was  a  very  diverse 
group:  the  10  recruited  via  flyers  were  all  students,  ranging  from  undergraduates  to  a  post¬ 
doctoral  student;  the  neighbors  both  had  college  degrees  but  no  current  academic 
connection.  (None  of  the  general  participants  had  a  degree  in  library  science,  and  none  was 
employed  as  a  librarian.)  However,  these  people  were  much  younger  than  the  librarians: 
except  for  a  woman  in  her  forties,  who  was  assigned  to  group  5  and  hence  will  not  affect  the 
official  NIST  results,  the  oldest  participant  was  in  their  thirties.  Five  were  women  and  seven 
were  men.  These  people  had  far  less  experience  with  online  searching  than  the  librarians:  one 
had  no  such  experience,  while  the  most  experienced  had  5  years  and  the  average  was  2.75. 
They  had  significantly  less  experience  than  the  librarians  in  searching  CDROMs  and 
commercial  services,  and  with  ranked  lists  and  relevance  feedback. 

See  Appendix  2  for  detailed  characteristics  of  the  searchers. 

Participants  were  told  that  the  study  would  take  about  3-1/2  hours  and  that  they  would  be  paid 
$35  if  they  completed  it.  While  20  people  completed  the  study,  one  general  participant  did 
not:  as  a  result  of  system  problems,  the  experimenter  ended  his  session  early.  The  incomplete 
data  we  gathered  for  this  participant  is  not  included  here. 


Group 

Type 

Control 

Experimental 

Size 

1 

General 

ZP 

AI 

4 

2 

Librarian 

ZP 

A1 

4 

3 

General 

ZP 

AI+ 

4 

4 

Librarian 

ZP 

AI+ 

4 

5 

General 

A1 

AI+ 

4 

Table  1.  Experimental  Setup 
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Procedures 


The  experiment  was  run  in  our  usability  laboratory  on  the  University  of 
Massachusetts/Amherst  campus.  A  “facilitator”  was  in  the  room  with  the  participant  all  of  the 
time  except  while  the  participant  was  doing  the  tutorials;  the  procedure  the  facilitator  followed 
is  given  by  the  checklist  in  Appendix  1.  The  same  person  acted  as  facilitator  for  all 
participants  except  for  the  last  two  in  group  5. 

First,  each  participant  filled  out  a  questionnaire  to  give  us  basic  demographic  information 
(age,  gender,  degrees,  general  computer  experience,  experience  with  various  types  of 
searching,  etc.).  Each  participant  also  took  two  standard  psychometric  tests  from  ETS  [4]:  a 
test  of  verbal  fluency  (Controlled  Associations,  test  FA-1),  and  a  test  of  structural  visualization 
(Paper  Folding,  test  VZ-2). 

Next,  the  participant  was  given  a  tutorial  to  learn  one  system,  then  they  worked  on  the  first 
three  topics.  After  a  short  break  they  were  given  a  tutorial  on  another  system,  then  they 
worked  on  the  other  three  topics.  Each  search  had  a  20-minute  time  limit,  and  the  participant 
was  instructed  to  stop  the  search  if  they  had  not  finished  in  20  minutes. 

We  gave  each  participant  a  piece  of  scratch  paper  before  each  search,  and  a  short 
questionnaire  after  each.  After  all  the  searches  were  finished  the  participant  was  given  a  final 
questionnaire,  and  then  “debriefed”.  The  study  was  conducted  single  blind:  the  participants 
were  not  told  until  the  debriefing  which  system  was  the  control  and  which  was  the 
experimental  system. 

We  ran  each  participant  through  the  entire  study  in  a  single  essentially  continuous  period  of 
slightly  over  three  to  slightly  over  four  hours,  with  no  breaks  longer  than  about  15  minutes. 

Grading  the  Tests 

The  Paper  Folding  test  is  multiple  choice,  and  grading  it  posed  no  problems.  However,  the 
Controlled  Associations  test  asks  the  subject  to  come  up  with  a  list  of  words  with  meanings 
“the  same  as  or  similar  to”  a  given  word:  clearly,  this  is  an  open-ended  question.  For  each  of 
the  given  words,  ETS  supplies  a  list  of  from  22  to  65  acceptable  words,  but  participants  used 
many  additional  words.  To  score  these,  first,  four  persons  (the  present  authors  and  the  wife  of 
one  author)  separately  judged  which  of  the  additional  words  should  be  accepted.  Then  the 
present  authors  met  and  reviewed  the  four  lists  of  acceptable  words.  Where  three  or  all  four 
judges  agreed,  we  followed  the  unanimous  or  majority  opinion;  where  two  judges  would 
accept  a  term  and  two  would  not,  we  discussed  it  and  reached  a  consensus  one  way  or  the 
other. 

To  obtain  the  list  of  additional  words  we  accepted  for  each  given  word,  contact  one  of  the 
authors. 
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Appendix  1.  Facilitator’s  Checklist 

Before  each  participant  comes: 

—  Consider  sending  e-mail  to  all  IR  lab  people,  asking  them  to  go  easy  on  the  test  machine. 

—  Make  sure  the  test  room  is  properly  set  up  and  reasonably  neat. 

—  Make  sure  the  computer  is  ready:  login  to  the  test  computer. 

—  Make  sure  the  camcorder  and  cassette  recorder  are  on  and  each  contains  a  properly- 

positioned  and  labelled  tape.  Check  that  the  computer’s  entire  screen  is  visible  thru 
the  camcorder’s  lens. 

—  Make  sure  the  documentation  is  in  place:  tutorials  for  ZPRISE,  Asplnquery, 

Asplnquery  Plus;  task  instructions;  search  topics;  sample  payment  forms. 

—  Have  a  pad  and  pens/pencils  for  taking  notes,  plus  a  stopwatch. 

—  Have  an  ink  pen  ready  for  participant  to  sign  the  consent  form. 

—  Check  the  folder  for  participant.  It  should  contain: 

—  Payment  forms 
—  Form  1.  informed  consent  form 
—  Form  2.  entry  (“PRE-SEARCH”)  questionnaire 
—  Controlled  Association  test 
—  Paper  Folding  test 
—  Form  3.  interview  form 
—  Form  4.  six  (6)  searcher  (aspect)  worksheets 
—  Form  5.  six  (6)  post-search  questionnaires 

—  Form  6/6p.  exit  questionnaire  (version  for  AI  or  AIP,  as  appropriate) 

At  the  beginning  of  each  test  session: 

—  Greet  participant;  check  their  name  to  be  sure  they’re  the  one  we  expect. 

—  Show  participant  where  to  sit. 

—  Make  participant  comfortable.  Offer  coffee,  juice,  snacks. 

—  Show  participant  the  video  camera  and  cassette  recorder.  Tell  them 

someone  will  be  watching  the  video  live  and  may  make  a  few  comments. 

—  Ask  participant  to  fill  out  both  payment  forms.  Tell  them  they  may  not  be 

paid  for  a  few  weeks. 

—  Ask  participant  to  read  and  sign  the  informed  consent  form. 

—  Ask  participant  to  fill  out  the  entry  questionnaire. 

—  Give  participant  the  Controlled  Associations  test  (6  min./page). 

—  Give  participant  the  Paper  Folding  test  (3  min./page). 

—  Give  participant  the  task  instructions. 

—  Give  participant  the  interview  form. 

—  Start  zprise,  Asplnquery,  or  Asplnquery  Plus  (commands  are  respectively  zp, 

ai,  aip);  if  zprise,  also  connect  to  the  database. 

—  Move  participant  to  the  computer,  and  give  them  the  tutorial  for  the  first  system. 
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—  Have  participant  click  the  appropriate  reset  button  (in  zprise,  “Stop  search, 

clear  results”;  in  our  systems,  “Reset”). 

—  Ask  if  participant  has  any  questions  (or  wants  a  break). 

—  Remind  participant  to  think  out  loud,  including  saying  what  they  like  or 

dislike;  tell  them  why  we  want  them  to  do  this. 

—  Remind  participant  to  wait  for  you  to  say  when  to  start  the  first  task,  to 

tell  you  when  he  or  she  has  finished  the  first  task,  and  then  to  wait  for 
you  to  say  when  to  start  the  second  task. 

—  Tell  participant  not  to  write  on  the  list  of  topics,  and  offer  them  a  worksheet. 
For  n  =  1  to  6  do: 

—  Check  time  left  on  videotape;  if  less  than  25  min.,  change  it. 

—  If  n=l  or  4,  start  the  camcorder. 

—  Offer  participant  a  worksheet  (unless  they  have  a  clean  one  from  a 
previous  search;  in  that  case,  remind  them  that  they  have  one). 

—  Have  participant  start  the  next  search. 

Allow  20  minutes  for  each  search.  At  the  end: 

—  Have  participant  click  the  appropriate  reset  button  (in  zprise,  “Stop 
search,  clear  results”;  in  our  systems,  “Reset”). 

—  Ask  participant  to  fill  out  a  post-search  questionnaire. 

—  If  n  =  3,  stop  the  camcorder;  switch  to  the  second  test  system  and 
give  participant  the  tutorial  for  it. 

End  do. 

—  Stop  the  camcorder.  Start  the  cassette  recorder. 

—  Ask  participant  to  fill  out  the  exit  questionnaire. 

—  Thank  participant  for  his  or  her  help. 

—  Debrief  participant.  Go  over  their  responses  to  the  exit  questionnaire. 

—  Thank  participant,  and  show  him  or  her  out. 

After  participant  leaves: 

—  Stop  the  cassette  recorder. 

—  Put  all  the  forms  in  participant’s  folder. 

—  Check  that  the  video  and  audio  tapes  are  properly  labelled  and  put  them  away. 


7 


Appendix  2.  Detailed  Characteristics  of  Participants 

The  following  is  a  summary  of  the  participants’  responses  to  the  Entry  questionnaire. 

A.  General  information.  For  Education,  we  show  only  the  highest  level  achieved  by  each 
participant. 


Librarians 

General 

Total 

Education: 

Undergraduate 

- 

6 

6 

Bachelor’s  degree 

- 

1 

1 

Master’s  student 

- 

1 

1 

Master’s  degree 

7 

2 

9 

Doctoral  student 

- 

1 

1 

Doctoral  degree 

1 

1 

2 

Age: 

Under  21 

_ 

3 

3 

21-30 

1 

7 

8 

31-40 

_ 

1 

1 

41-50 

5 

1 

6 

51-60 

2 

_ 

2 

Over  60 

- 

- 

- 

Male/Female 

2  Male  /  6  Female 

7  Male  /  5  Female 

9  Male  /II  Female 

B.  Computer  and  searching  experience.  All  figures  given  are  means 

followed  by  medians 

(“mean,  median’’).  Except  for  “Years  searching’’, 

all  are  on  a  scale  of  1  to  5,  with  1  =  none, 

5  =  a  lot. 

Librarians 

General 

Total 

Computer  usage 

3.63,  3 

3.92,  4 

3.8,  4 

Years  searching 

10.0,  8.5 

2.75,  2.25 

5.65,  4 

Search  library  catalogs 

4.63,  5 

3.67,  4 

4.05,  4 

Search  CDROMs 

3.88,  4.5 

2.42,  2 

3.0,  3 

Search  commercial  services 

3.63,  4 

1.33,  1 

2.25,  2 

Search  the  WWW 

3.5,  3.5 

3.75,  4 

3.65,  4 

Search  other 

2.0,  1 

1.57,  1 

1.73,  1 
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Full-text  databases 

3.0,  3 

2.75,  2.5 

2.85,  3 

Ranked  output 

2.75,  3 

1.73,  1 

2.16,  2 

Relevance  feedback 

2.38,  2.5 

1.58,  1 

1.9,  1.5 

Mouse-based  interface 

3.63,  3.5 

4.42,  4.5 

4.1,  4 

3-D  interfaces 

1,  1 

2.1,  1.5 

1.65,  1 
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Appendix  3.  Test  Scores 

The  following  is  a  summary  of  the  participants’  responses  to  the  psychometric  tests.  Figures 
given  are  means  followed  by  medians  (“mean,  median”). 


Librarians  General 

Controlled  Associations  (FA-1)  38.125,  35.5  27.25,  26.5 

Paper  Folding  (VZ-2)  9.375,9.125  10.79,12 


Total 
31.6,  32 
10.23,  10.88 
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Appendix  4.  Protocol  for  One  Topic 

Following  is  the  log  of  the  interaction  for  Participant  13,  Topic  1  (326i).  Spoken  words  are 
shown  in  italics.  “U:”  (for  “User”)  precedes  remarks  and  actions  by  the  participant;  “F:” 
precedes  remarks  by  the  Facilitator. 

Time  set  at  Tue  Aug  12  09:24:57  1997 
query  is  ferry  sinking  casualties 
Query  is:  ferry  sinking  casualties 

bl->term_freq  =  0,  default_belief  =  0.400000,  totalhits  =  2932 

bl->doc_cnt  =  20 

24  s  Tue  Aug  12  09:25:21  1997 

Number  of  docs  found  is  20 

1:  177935:  FT943-312:  FT  30  SEP  94  /  Ferries  in  six  'near  accidents':  Finland  and  Sweden  order  checks 
after  Estonia  sinking 

2:  205199:  FT944- 15661:  FT  17  OCT  94  /  World  News  in  Brief:  Bangladesh  ferry  sinks 

3:  174281:  FT943-178:  FT  30  SEP  94  /  Leading  Article:  Defying  the  cruel  sea 

4:  204595:  FT944- 15057:  FT  20  OCT  94  /  Improved  ferry  safety  urged 

5:  194241 :  FT944-5773:  FT  02  DEC  94  /  World  News  in  Brief:  Manila  ferry  sinks 

6:  200503:  FT944-1 1367:  FT  07  NOV  94  /  Pounds  45m  car-ferry  research  planned 

7:  199238:  FT944-10102:  FT  12  NOV  94  /  Tighter  ferry  rules  proposed 

8:  2081 1 1 :  FT944-18217:  FT  05  OCT  94  /  World  News  in  Brief:  Check  on  ferries  ordered 

9:  200184:  FT944-11048:  FT  08  NOV  94  /  Bow  doors  faulty  on  33%  of  ferries  using  UK  ports: 

Government  to  increase  safety  checks  on  vessels 

10:  208769:  FT944-18875:  FT  01  OCT  94  /  What  future  for  the  ferry?:  Questions  raised  by  the  Baltic 
tragedy 

11:  178166:  FT943-543:  FT  29  SEP  94  /  Bow  doors  leak  reported  after  800  die  in  Baltic  ferry  sinking 

12:  193552:  FT944-5084:  FT  06  DEC  94  /  Ro-ro  ferry  study  agreed 

13:  75524:  FT931-8485:  FT  19  FEB  93  /  Crowded  ferry  sinks  off  Haiti 

14:  199245:  FT944-10109:  FT  12  NOV  94  /  Tighter  ferry  rules  proposed 

15:  193716:  FT944-5248:  FT  05  DEC  94  /  Sea  safety  review  focuses  on  ferries 

16:  178159:  FT943-536:  FT  29  SEP  94  /  Safety  rules  that  failed  the  Estonia:  it  was  a  modern  ship,  well 
maintained  and  partly  Swedish  owned.  But  are  even  the  best  ro-ro  ferries  vulnerable? 

17:  177939:  FT943-316:  FT  30  SEP  94  /  Ferries  face  calls  for  safety  curbs:  Estonia  disaster  brings 
reports  of  other  'near  accidents' 

18:  199989:  FT944-10853:  FT  09  NOV  94  /  Eurotunnel  hits  at  government  on  ferry  safety 
19:  39232:  FT923-4546:  FT  05  SEP  92  /  Swan  wins  order  for  Tyne  ferry 
20:  39203:  FT923-4517:  FT  05  SEP  92  /  Ferry  order  for  Tyne  yard 

38  s,  Reading  doc  177935:FT943-312,  click  from  main  win,  time  Tue  Aug  12  09:25:35  1997 

0:42  U:  OK,  so  my  first  article  is  about  a  ferry  that  sank  and  900  people  died. 

71  s,  Reading  doc  177935:FT943-312,  click  from  main  win,  time  Tue  Aug  12  09:26:08  1997 
Doc  number  177935  added  to  aspect  0 

Aspect  #  0,  auto  terms  are  estonia  "estonia  sink"  forsberg  "estonia  disaster"  "bow  section" 

No  user  supplied  text 

1:18  U :This  article  describes  six  incidents. 

1:30  F:  Six  incidents  of  ferry  sinkings? 

U:  Right. 


1  1 


1:48  U:  Oh,  talks  about  six  near  accidents  and  it  describes  one  that  actually  happened. 

148  s,  Reading  doc  205199:FT944-15661,  click  from  main  win,  time  Tue  Aug  12  09:27:25  1997 
Doc  number  205199  added  to  aspect  1 

2:30  U:  This  is  a  brief  article  about  400  people  that  dies  in  Bangladesh  so  we’ll 
save  that,  and  the  name  ...  we’ll  name  that  one  Bangladesh. 

Aspect  #  1,  auto  terms  are  "ferry  sink"  "ferry  disaster"  "wedding  party"  "high  sea"  bangladesh 
U:  drags  "bangladesh"  into  label  area 

197  s,  Reading  doc  174281:FT943-178,  click  from  main  win,  time  Tue  Aug  12  09:28:14  1997 

3: 1 8  U:  Here’s  another  article  about  the  Estonia  incident.  It ’s  a  repeat  so  I  don 't 

need  to  save  that,  or  should  I  save  that  also  under 

F:  No,  there’s  no  need,  there’s  no  point  to  saving  additional  ones. 

224  s,  Reading  doc  204595:FT944-15057,  click  from  main  win,  time  Tue  Aug  12  09:28:41  1997 

3:45  U:  This  one  just  talks  about  the  Estonia  again,  so  we  don’t  need  that. 

232  s,  Reading  doc  194241  :FT944-5773,  click  from  main  win,  time  Tue  Aug  12  09:28:49  1997 

3:53  U:  And  the  fifth  one  is  in  Manila,  480  people  were  on  it,  275  were  rescued, 
and  they’re  still  picking  up  survivors,  so  you  can  probably  assume  100  people 
died,  so  go  ahead  and  save  it. 

Doc  number  194241  added  to  aspect  2 

Aspect  #  2,  auto  terms  are  "ferry  sink"  "cargo  ship"  manila  sink  survivor 
U:  drags  "manila"  into  label  area 

267  s,  Reading  doc  200503:FT944-1 1367,  click  from  main  win,  time  Tue  Aug  12  09:29:24  1997 

272  s,  Reading  doc  199238:FT944-10102,  click  from  main  win,  time  Tue  Aug  12  09:29:29  1997 

4:34  U:  A  lot  of  these  keep  talking  about  tighter  regulations  due  to  the  sinking  of 
the  Estonia. 

288  s,  Reading  doc  200 184:FT944-1 1048,  click  from  main  win,  time  Tue  Aug  12  09:29:45  1997 
Doc  number  200184  added  to  aspect  3 

Aspect  #  3,  auto  terms  are  "bow  door"  "marine  safety  agent"  ferry  "safety  agent"  "dr  mawhinney" 
4:57  U:  Here’s  one  that  briefly  mentions  a  ship... 

5:09  U:  This  one  again  is  more  about  safety  regulations,  but  it  briefly  mentions  a 
ship  that  had  193  casualties,  so  I  guess  I’ll  type  in  my  own  word  since  the  one 
I  want  isn  't  in  there. 

U:  types  "  Flerald  of  Free  Enterprise"  in  label  area. 

5:38  U:  I’m  trying  to  name  all  these  either  by  the  name  of  the  ship  or  where  it  happened. 
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U:  moves  controls  on  3-D  window  and  alters  view  several  times. 


5:54  U:  I’m  trying  to  see  if  I  can  use  the  3-D  to  help  me  out 

F:  I’m  sorry,  you  couldn't  couldn’t  what. ..You’re  trying  to  see  if 

U:  I'm  trying  to  see  if  I  can  use  this  to  give  me  ...  I'm  assuming  that  these 

are  supposed  to  show  articles  in  the  connecting  blocks  that  are  more  relevant 

F:  That  are  more  similar  to  each  other 

U:  More  similar. 

407  s,  Reading  doc  208769:FT944-18875,  click  from  main  win,  time  Tue  Aug  12  09:31:44  1997 

421  s,  Reading  doc  178166:FT943-543,  click  from  main  win,  time  Tue  Aug  12  09:31:59  1997 

448  s,  Reading  doc  193552:FT944-5084,  click  from  main  win,  time  Tue  Aug  12  09:32:25  1997 

454  s,  Reading  doc  75524:FT93 1-8485,  click  from  main  win,  time  Tue  Aug  12  09:32:31  1997 
Doc  number  75524  added  to  aspect  4 

Aspect  #  4,  auto  terms  are  "ferry  sink"  port-au-prince  neptune  haiti  "product  centre" 

U:  drags  "neptune"  into  label  area 

485  s,  Reading  doc  199245:FT944-10109,  click  from  main  win,  time  Tue  Aug  12  09:33:02  1997 

491  s,  Reading  doc  193716:FT944-5248,  click  from  main  win,  time  Tue  Aug  12  09:33:08  1997 

493  s,  Reading  doc  177939:FT943-316,  click  from  main  win,  time  Tue  Aug  12  09:33:10  1997 

500  s,  Reading  doc  199989:FT944-10853,  click  from  main  win,  time  Tue  Aug  12  09:33:17  1997 

505  s,  Reading  doc  39232:FT923-4546,  click  from  main  win,  time  Tue  Aug  12  09:33:22  1997 

512  s,  Reading  doc  39203:FT923-4517,  click  from  main  win,  time  Tue  Aug  12  09:33:29  1997 

8:32  U:  So  I  went  through  all  20  of  the  articles.  For  the  most  part  I’d  say  all  but 
probably  3  or  4  talked  about  accidents  with  over  100  casualties,  so  should  I  try 
a  new  search? 

F:  It’s  up  to  you.  You  have  plenty  of  time. 

U:  You  mean  try  a  different  wording  of  it? 

F:  It's  up  to  you. 

9:20  F:  You  could  also  try  raising  the  max  docs. 

U:  OK. 

query  is  ferry  sinking  casualties 
Query  is:  ferry  sinking  casualties 

bl->term_freq  =  0,  default_belief  =  0.400000,  totalhits  =  2932 

bl->doc_cnt  =  40 

574  s  Tue  Aug  12  09:34:31  1997 

Number  of  docs  found  is  40 

1:  177935:  FT943-312:  FT  30  SEP  94  /  Ferries  in  six  'near  accidents':  Finland  and  Sweden  order  checks 
after  Estonia  sinking 

2:  205199:  FT944- 15661:  FT  17  OCT  94  /  World  News  in  Brief:  Bangladesh  ferry  sinks 
3:  174281:  FT943-178:  FT  30  SEP  94  /  Leading  Article:  Defying  the  cruel  sea 
4:  204595:  FT944- 15057:  FT  20  OCT  94  /  Improved  ferry  safety  urged 
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5:  194241 :  FT944-5773:  FT  02  DEC  94  /  World  News  in  Brief:  Manila  ferry  sinks 

6:  200503:  FT944-1 1367:  FT  07  NOV  94  /  Pounds  45m  car-ferry  research  planned 

7:  199238:  FT944-10102:  FT  12  NOV  94  /  Tighter  ferry  rules  proposed 

8 :  208 1 1 1 :  FT944- 18217:  FT  05  OCT  94  /  World  News  in  Brief:  Check  on  ferries  ordered 

9:  200184:  FT944-11048:  FT  08  NOV  94  /  Bow  doors  faulty  on  33%  of  ferries  using  UK  ports: 

Government  to  increase  safety  checks  on  vessels 

10:  208769:  FT944-18875:  FT  01  OCT  94  /  What  future  for  the  ferry?:  Questions  raised  by  the  Baltic 
tragedy 

11:  178166:  FT943-543:  FT  29  SEP  94  /  Bow  doors  leak  reported  after  800  die  in  Baltic  ferry  sinking 

12:  193552:  FT944-5084:  FT  06  DEC  94  /  Ro-ro  ferry  study  agreed 

13:  75524:  FT931-8485:  FT  19  FEB  93  /  Crowded  ferry  sinks  off  Haiti 

14:  199245:  FT944-10109:  FT  12  NOV  94  /  Tighter  ferry  rules  proposed 

15:  193716:  FT944-5248:  FT  05  DEC  94  /  Sea  safety  review  focuses  on  ferries 

16:  178159:  FT943-536:  FT  29  SEP  94  /  Safety  rules  that  failed  the  Estonia:  it  was  a  modern  ship,  well 
maintained  and  partly  Swedish  owned.  But  are  even  the  best  ro-ro  ferries  vulnerable? 

17:  177939:  FT943-316:  FT  30  SEP  94  /  Ferries  face  calls  for  safety  curbs:  Estonia  disaster  brings 
reports  of  other  'near  accidents' 

18:  199989:  FT944-10853:  FT  09  NOV  94  /  Eurotunnel  hits  at  government  on  ferry  safety 

19:  39232:  FT923-4546:  FT  05  SEP  92  /  Swan  wins  order  for  Tyne  ferry 

20:  39203:  FT923-4517:  FT  05  SEP  92  /  Ferry  order  for  Tyne  yard 

21:  207852:  FT944-17958:  FT  05  OCT  94  /  Finns  order  ro-ro  bow  doors  welded  shut 

22:  169325:  FT942-14757:  FT  19  APR  94  /  Letters  to  the  Editor:  Channel  control  overdue 

23:  207851:  FT944-17957:  FT  05  OCT  94  /  UN  maritime  agency  panel  to  review  safety:  A  look  at 

action  prompted  by  the  Baltic  ferry  disaster 

24:  208393:  FT944- 18499:  FT  04  OCT  94  /  Baltic  ferry  operators  to  weld  bow  doors  shut:  Safety  move 
follows  confirmation  of  cause  of  Estonia  disaster 

25:  208098:  FT944-18204:  FT  05  OCT  94  /  Maritime  agency  in  safety  plan 

26:  208402:  FT944- 18508:  FT  04  OCT  94  /  Estonia's  bow  doors  were  torn  off  in  heavy  storm:  Video  of 
sunken  ferry  shows  how  water  flooded  car  deck 

27:  204681:  FT944-15143:  FT  19  OCT  94  /  Estonia's  missing  bow  door  located 

28:  200149:  FT944-11013:  FT  08  NOV  94  /  International  Company  News:  Heavy  loss  in  US  pushes 

Trygg-Hansa  into  the  red  -  Swedish  insurer  posts  SKr813m  deficit  at  nine  months 

29:  141158:  FT941-5434:  FT  07  MAR  94  /  Freight  companies  to  shun  Channel  tunnel 

30:  208832:  FT944-18938:  FT  01  OCT  94  /  UN  agency  orders  ferry  probe:  Estonia's  bow  doors  may 

have  been  torn  off  in  storm,  Swedish  authorities  say 

31:  178158:  FT943-535:  FT  29  SEP  94  /  Tragedy  leaves  Swedes  in  shock 

32:  171243:  FT942-16675:  FT  08  APR  94  /  Survey  of  East  Kent  (7):  Pain  amid  the  gain  -  The  ferries 
fight  back 

33:  195783:  FT944-6974:  FT  26  NOV  94  /  Thinking  the  unsinkable:  The  modern  parallels  exposed  by 
an  exhibition  about  the  Titanic,  which  sank  in  1912 

34:  205276:  FT944-15738:  FT  17  OCT  94  /  Company  News  This  Week:  Departure  delays  leave 
investors  counting  the  cost  -  Eurotunnel 

35:  137406:  FT934-1954:  FT  16  DEC  93  /  Technology:  Ships  bridge  the  danger  gap  -  Andrew  Fisher 
concludes  a  series  on  transport  safety  with  an  investigation  into  innovations  that  may  help  prevent  sea 
disasters  and  give  clues  to  their  causes 

36:  127095:  FT934-8445:  FT  16  NOV  93  /  Corporate  bankruptcies  increase  as  demand  sinks 

37:  1655:  FT91 1-4602:  FT  18  APR  91  /  MMC  to  investigate  Isle  of  Wight  ferries 

38:  206611:  FT944-1600:  FT  19  DEC  94  /  Survey  of  Sweden  (14):  A  remarkable  comeback  -  Profile: 

Stena  Line 

39:  26988:  FT922-7334:  FT  19  MAY  92  /  World  Trade  News:  Denmark-Sweden  ferry  link-up  is  agreed 
40:  1 19826:  FT933-1606:  FT  23  SEP  93  /  Ferry  operator  in  link  with  Belgium 

U:  Several  3-D  interactions 
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Reading  doc  200503:  FT944-1 1367,  click  from  3-D  window 

642  s,  Reading  doc  207852:FT944-17958,  click  from  main  win,  time  Tue  Aug  12  09:35:39  1997 

652  s,  Reading  doc  169325:FT942-14757,  click  from  main  win,  time  Tue  Aug  12  09:35:49  1997 

661  s,  Reading  doc  207851:FT944-17957,  click  from  main  win,  time  Tue  Aug  12  09:35:58  1997 

11:11  U:  So  I  increased  the  maxdocs  from  20  to  40,  and  most  of  the  later  articles 
don’t  seem  to  really  have  much  relevant  information.  Either  they’re  talking 
about  the  Estonia  or  they’re  just  talking  about  general  safety  regulations. 

678  s,  Reading  doc  208393:FT944-18499,  click  from  main  win,  time  Tue  Aug  12  09:36:15  1997 

701  s,  Reading  doc  208098:FT944-18204,  click  from  main  win,  time  Tue  Aug  12  09:36:38  1997 

11:45  U:  I’m  guessing  that’s  why  there’s  this  big  network  here.  (Points  to 
large  cluster  of  documents  in  3-D  viewer.)  A  lot  of  them  are 
talking  about  the  Estonia  so  I  think  they’re  all  related  in  that  sense. 

723  s,  Reading  doc  208402:FT944- 18508,  click  from  main  win,  time  Tue  Aug  12  09:37:00  1997 

734  s,  Reading  doc  204681:FT944-15143,  click  from  main  win,  time  Tue  Aug  12  09:37:1 1  1997 

737  s,  Reading  doc  200 149:FT944-1 1013,  click  from  main  win,  time  Tue  Aug  12  09:37:14  1997 

744  s,  Reading  doc  141 158:FT941-5434,  click  from  main  win,  time  Tue  Aug  12  09:37:21  1997 

12:32  U:  Yeah  this  is  really  starting  to  get ...  My  query  is  “ferry  sinking”,  and 

in  this  article  the  word  “sink”  only  appears  once,  and  it  doesn’t  have  anything 

to  do  with  ferries,  and  there’s  nothing  about  casualties  so  it  looks  like  we’re 

getting  farther  and  farther  away  from  anything  relevant.  You  can  see  that  over 

here,  we’re  moving  further  away  from  this  point.  (Points  to  several  documents  in  3-D  view) 

775  s,  Reading  doc  208832:FT944-18938,  click  from  main  win,  time  Tue  Aug  12  09:37:52  1997 

800  s,  Reading  doc  17 1243: FT942- 16675,  click  from  main  win,  time  Tue  Aug  12  09:38:17  1997 

813  s,  Reading  doc  195783:FT944-6974,  click  from  main  win,  time  Tue  Aug  12  09:38:30  1997 

822  s,  Reading  doc  205276:FT944-15738,  click  from  main  win,  time  Tue  Aug  12  09:38:39  1997 

827  s,  Reading  doc  137406:FT934-1954,  click  from  main  win,  time  Tue  Aug  12  09:38:44  1997 

13:49  U:  OK,  I’ve  found  a  new  one. 

Doc  number  137406  added  to  aspect  5 

Aspect  #  5,  auto  terms  are  moby  imo  vessel  livorno  ship 

U:  drags  "livorno"  into  label  area 

14:06  U:  This  is  the  first  new  article  I’ve  found  in  the  last  20  I’ve  looked  at. 

868  s,  Reading  doc  127095:FT934-8445,  click  from  main  win,  time  Tue  Aug  12  09:39:25  1997 
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872  s,  Reading  doc  1 655 :FT91 1-4602,  click  from  main  win,  time  Tue  Aug  12  09:39:29  1997 
876  s,  Reading  doc  20661 1:FT944-1600,  click  from  main  win,  time  Tue  Aug  12  09:39:33  1997 
884  s,  Reading  doc  26988:FT922-7334,  click  from  main  win,  time  Tue  Aug  12  09:39:41  1997 
892  s,  Reading  doc  1 19826:FT933-1606,  click  from  main  win,  time  Tue  Aug  12  09:39:49  1997 
15:00  F:  You  have  five  minutes. 


15:23  U:  We’ll  try  searching  for  ferry  and  accidents. 


query  is  feny  accident 
Query  is:  ferry  accident 

bl->term_freq  =  0,  default_belief  =  0.400000,  totalhits  =  1978 

bl->doc_cnt  =  40 

932  s  Tue  Aug  12  09:40:29  1997 

Number  of  docs  found  is  40 

1:  174533:  FT943-3295:  FT  15  SEP  94  /  Inquiry  starts  after  six  die  in  ferry  walkway  collapse 

2:  42744:  FT923-7671:  FT  15  AUG  92  /  Deaths  ferry  to  be  withdrawn 

3:  149044:  FT94 1-12581:  FT  29  JAN  94  /  Accident  halts  ferry  services 

4:  72637:  FT931-5947:  FT  03  MAR  93  /  World  News  in  Brief:  Congo  ferry  toll  rises  to  146 

5:  177935:  FT943-312:  FT  30  SEP  94  /  Ferries  in  six  'near  accidents':  Finland  and  Sweden  order  checks 


after  Estonia  sinking 

6:  186187:  FT943-1246:  FT  26  SEP  94  /  World  News  in  Brief:  16  injured  in  lifeboat  accident 
7:  208393:  FT944-18499:  FT  04  OCT  94  /  Baltic  ferry  operators  to  weld  bow  doors  shut:  Safety  move 
follows  confirmation  of  cause  of  Estonia  disaster 

8:  208402:  FT944-18508:  FT  04  OCT  94  /  Estonia's  bow  doors  were  torn  off  in  heavy  storm:  Video  of 
sunken  ferry  shows  how  water  flooded  car  deck 

9:  9804:  FT92 1-686:  FT  27  MAR  92  /  Crash  probe  finds  'no  abnormality' 

10:  174478:  FT943-3240:  FT  15  SEP  94  /  Investigators  widen  probe  on  ferry  walkway  collapse 

11:  186180:  FT943-1239:  FT  26  SEP  94  /  World  News  in  Brief:  16  injured  in  lifeboat  accident 

12:  207852:  FT944-17958:  FT  05  OCT  94  /  Finns  order  ro-ro  bow  doors  welded  shut 

13:  177939:  FT943-316:  FT  30  SEP  94  /  Ferries  face  calls  for  safety  curbs:  Estonia  disaster  brings 

reports  of  other  'near  accidents' 

14:  201958:  FT944-12822:  FT  31  OCT  94  /  Business  Travel:  In  S  Korea,  it  is  better  to  arrive  .. 

15:  208769:  FT944-18875:  FT  01  OCT  94  /  What  future  for  the  ferry?:  Questions  raised  by  the  Baltic 


tragedy 

16:  14222:  FT921-1 1074:  FT  03  FEB  92  /  UK  Company  News:  Eurotunnel  to  seek  damages  for  cost  of 
extra  safety 

17:  178552:  FT943-6917:  FT  26  AUG  94  /  Cross-Channel  ferry  blaze  to  be  investigated 
18:  1655:  FT91 1-4602:  FT  18  APR  91  /  MMC  to  investigate  Isle  of  Wight  ferries 
19:  5733:  FT921-365:  FT  30  MAR  92  /  Hopes  for  ship  data  recorder 

20:  26988:  FT922-7334:  FT  19  MAY  92  /  World  Trade  News:  Denmark-Sweden  ferry  link-up  is  agreed 
21:  119826:  FT933-1606:  FT  23  SEP  93  /  Ferry  operator  in  link  with  Belgium 

22:  150782:  FT941-1125:  FT  26  MAR  94  /  International  Company  News:  Vard  plans  to  spin  off  ferry 
division 

23:  118260:  FT933-15867:  FT  07  JUL  93  /  New  high-speed  Stena  ferry  in  service  by  1995 

24:  1 19845:  FT933-1625:  FT  23  SEP  93  /  Sally  Line  agrees  Belgian  link-up 

25:  32757:  FT922- 12800:  FT  15  APR  92  /  Freight  ferry 

26:  199245:  FT944-10109:  FT  12  NOV  94  /  Tighter  ferry  rules  proposed 

27:  114042:  FT933-11894:  FT  27  JUL  93  /  International  Company  News:  Vard  set  to  spin  off  ferry  unit 
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28:  200184:  FT944-11048:  FT  08  NOV  94  /  Bow  doors  faulty  on  33%  of  ferries  using  UK  ports: 
Government  to  increase  safety  checks  on  vessels 

29:  62351 :  FT924-1 1264:  FT  27  OCT  92  /  Ferry  operators  accused  of  pricing  collusion 
30:  199989:  FT944-10853:  FT  09  NOV  94  /  Eurotunnel  hits  at  government  on  ferry  safety 
31:  143053:  FT941-732:  FT  29  MAR  94  /  Netherlands  ferry  route  may  restart 
32:  199238:  FT944-10102:  FT  12  NOV  94  /  Tighter  ferry  rules  proposed 
33:  84325:  FT931-16573:  FT  06  JAN  93  /  Cross-Channel  ferries  hint 

34:  125074:  FT934-497:  FT  24  DEC  93  /  International  Company  News:  Greek  ferry  operator  in  cash  call 

35:  64622:  FT924-13535:  FT  15  OCT  92  /  New  ferry  service 

36:  28865:  FT922-921 1:  FT  08  MAY  92  /  New  ferry  is  largest  in  Channel 

37:  137406:  FT934-1954:  FT  16  DEC  93  /  Technology:  Ships  bridge  the  danger  gap  -  Andrew  Fisher 
concludes  a  series  on  transport  safety  with  an  investigation  into  innovations  that  may  help  prevent  sea 
disasters  and  give  clues  to  their  causes 

38:  2421:  FT91 1-5368:  FT  15  APR  91  /  World  News  in  Brief:  Ferries  disrupted 
39:  26728:  FT922-7074:  FT  20  MAY  92  /  Boulogne  freight  link 
40:  44107:  FT923-9034:  FT  07  AUG  92  /  Ferry  row  settled 

951  s,  Reading  doc  174533:FT943-3295,  click  from  main  win,  time  Tue  Aug  12  09:40:48  1997 

956  s,  Reading  doc  42744:FT923-7671,  click  from  main  win,  time  Tue  Aug  12  09:40:53  1997 

965  s,  Reading  doc  149044:FT941-12581,  click  from  main  win,  time  Tue  Aug  12  09:41:02  1997 

970  s,  Reading  doc  72637:FT931-5947,  click  from  main  win,  time  Tue  Aug  12  09:41:07  1997 
Doc  number  72637  added  to  aspect  6 

Aspect  #  6,  auto  terms  are  Congo  brazzaville  zairean  "illegal  immigrant"  "death  toll" 

U:  drags  "congo"  into  label  area 

992  s,  Reading  doc  208393:FT944-18499,  click  from  main  win,  time  Tue  Aug  12  09:41:29  1997 
1006  s,  Reading  doc  208402:FT944- 18508,  click  from  main  win,  time  Tue  Aug  12  09:41:43  1997 
1008  s,  Reading  doc  9804:FT92 1-686,  click  from  main  win,  time  Tue  Aug  12  09:41:45  1997 
1012  s,  Reading  doc  186180:FT943-1239,  click  from  main  win,  time  Tue  Aug  12  09:41:49  1997 
1016  s,  Reading  doc  207852:FT944-17958,  click  from  main  win,  time  Tue  Aug  12  09:41:53  1997 
1020  s,  Reading  doc  177939:FT943-316,  click  from  main  win,  time  Tue  Aug  12  09:41:57  1997 
1022  s,  Reading  doc  201958:FT944-12822,  click  from  main  win,  time  Tue  Aug  12  09:41:59  1997 
1031  s,  Reading  doc  208769:FT944- 18875,  click  from  main  win,  time  Tue  Aug  12  09:42:08  1997 
1034  s,  Reading  doc  14222:FT921-1 1074,  click  from  main  win,  time  Tue  Aug  12  09:42:11  1997 
1041  s,  Reading  doc  178552:FT943-6917,  click  from  main  win,  time  Tue  Aug  12  09:42:18  1997 
1048  s,  Reading  doc  1 655 :FT91 1-4602,  click  from  main  win,  time  Tue  Aug  12  09:42:25  1997 
1054  s,  Reading  doc  5733:FT921-365,  click  from  main  win,  time  Tue  Aug  12  09:42:31  1997 
1064  s,  Reading  doc  26988:FT922-7334,  click  from  main  win,  time  Tue  Aug  12  09:42:41  1997 
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1070  s,  Reading  doc  1 19826:FT933-1606,  click  from  main  win,  time  Tue  Aug  12  09:42:47  1997 
1079  s,  Reading  doc  1 18260:FT933-15867,  click  from  main  win,  time  Tue  Aug  12  09:42:56  1997 
1082  s,  Reading  doc  1 19845:FT933-1625,  click  from  main  win,  time  Tue  Aug  12  09:42:59  1997 
1089  s,  Reading  doc  32757:FT922-12800,  click  from  main  win,  time  Tue  Aug  12  09:43:06  1997 
1092  s,  Reading  doc  199245:FT944-10109,  click  from  main  win,  time  Tue  Aug  12  09:43:09  1997 
1096  s,  Reading  doc  1 14042:FT933-1 1894,  click  from  main  win,  time  Tue  Aug  12  09:43:13  1997 
1105  s,  Reading  doc  125074:FT934-497,  click  from  main  win,  time  Tue  Aug  12  09:43:22  1997 
U:  does  extensive  interactions  with  3-D  window 

19:06  F:  Could  you  say  what  you’re  doing  there?  With  the  3-D  window? 

U:  I’m  just  looking  at  it  and  trying  to  see  how  the  articles  I’ve  picked  lay  out 
in  this  3-D  netM’ork.  I’m  just  trying  to  figure  out  how  I  could  make  it  more 
useful  for  my  searching  purposes.  I'm  really  thinking  about  things  how  if  I’m 
searching  for  things  on  the  Internet  and  I  had  something  like  this  how  would  I 
be  able  to  use  it.  It’s  an  interesting  idea. 

20:00  F:  Time’s  up. 

7  documents  saved  in  7  aspects,  with  0  miscellaneous  docs 
Aspect  0,  1  docs  saved 

Auto  Terms:  estonia  estonia_sink  forsberg  estonia_disaster  bow_section 
User  supplied  text  =  estonia 

FT943-312:  177935  FT  30  SEP  94  /  Ferries  in  six  'near  accidents':  Finland  and  Sweden  order  checks  after 
Estonia  sinking 

Aspect  1,  1  docs  saved 

Auto  Terms:  ferry _sink  ferry _disaster  wedding_party  high_sea  bangladesh 
User  supplied  text  =  bangladesh 

FT944-15661:  205199  FT  17  OCT  94  /  World  News  in  Brief:  Bangladesh  ferry  sinks 
Aspect  2,  1  docs  saved 

Auto  Terms:  ferry _sink  cargo_ship  manila  sink  survivor 
User  supplied  text  =  manila 

FT944-5773:  194241  FT  02  DEC  94  /  World  News  in  Brief:  Manila  ferry  sinks 
Aspect  3,  1  docs  saved 

Auto  Terms:  bow_door  marine_safety_agent  ferry  safety _agent  dr_mawhinney 
User  supplied  text  =  Herald  of  Free  Enterprise 

FT944-1 1048:  200184  FT  08  NOV  94  /  Bow  doors  faulty  on  33%  of  ferries  using  UK  ports:  Government 
to  increase  safety  checks  on  vessels 

Aspect  4,  1  docs  saved 

Auto  Terms:  ferry_sink  port-au-prince  neptune  haiti  product_centre 
User  supplied  text  =  neptune 

FT93 1-8485:  75524  FT  19  FEB  93  /  Crowded  ferry  sinks  off  Haiti 
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Aspect  5,  1  docs  saved 

Auto  Terms:  moby  imo  vessel  livorno  ship 

User  supplied  text  =  livorno 

FT934-1954:  137406  FT  16  DEC  93  /  Technology:  Ships  bridge  the  danger  gap  -  Andrew  Fisher 
concludes  a  series  on  transport  safety  with  an  investigation  into  innovations  that  may  help  prevent  sea 
disasters  and  give  clues  to  their  causes 

Aspect  6,  1  docs  saved 

Auto  Terms:  Congo  brazzaville  zairean  illegal_immigrant  death_toll 
User  supplied  text  =  Congo 

FT931-5947:  72637  FT  03  MAR  93  /  World  News  in  Brief:  Congo  ferry  toll  rises  to  146 


1200  s  Tue  Aug  12  09:45:21  1997 
Stats  from  this  run:  3  queries  run 
100  docs  returned,  66  unique,  52  viewed 
7  docs  saved  (including  misc),  7  saved 


saved  docs: 

FT93 1-5947:  72637  979 
FT931-8485:  75524  466 
FT934-1954:  137406  843 
FT943-312:  177935  75 
FT944-5773:  194241  255 
FT944- 11048:  200184  311 
FT944- 15661:  205199  162 


saved  good  docs 
FT93 1-5947:  72637  979 
FT931-8485:  75524  466 
FT934-1954:  137406  843 
FT943-312:  177935  75 
FT944-5773:  194241  255 
FT944- 11048:  200184  311 
FT944- 15661:  205199  162 


Sparse  Tree  Data  Starts  HERE 

1  FT943-312 

2  FT944- 15661 

3  FT944-5773 

4  FT944-1 1048 

5  FT93 1-8485 

6  FT934-1954 

7  FT93 1-5947 
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Appendix  5.  Experimental  Materials 

The  following  materials  were  used  with  participants. 

Form  1.  Informed  Consent  Form 

Reprinted  below. 

Form  2.  Entry  (“PRE- SEARCH”)  questionnaire 

Reprinted  below. 

Controlled  Association  test 

This  is  test  FA-1  in  [4];  to  obtain  a  copy,  contact  ETS. 

Paper  Folding  test 

This  is  test  VZ-2  in  [4];  to  obtain  a  copy,  contact  ETS. 

Form  3.  Interview  form 

Reprinted  below. 

Form  4.  Searcher  (aspect)  worksheet 

Reprinted  below. 

Form  5.  Post-search  questionnaire 

Reprinted  below. 

Form  6.  Exit  questionnaire 

There  were  two  versions,  one  for  participants  who  used  AI  and  ZPRISE,  and  one  for 
participants  who  used  AIP  and  ZPRISE.  The  AIP  version  is  reprinted  below;  the  AI  version 
was  identical  except  that  it  omits  the  page  of  questions  about  the  3-D  window. 
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UMASS  TREC-6  INTERACTIVE  SEARCHING  STUDY 
INFORMED  CONSENT  FORM 


The  University’s  Human  Studies  Research  Committee  has  approved  this  study  and  the  recruitment  of 
participants.  The  study  is  sponsored  by  the  National  Science  Foundation. 


PURPOSE  OF  STUDY  AND  DURATION 

The  purpose  of  this  study  is  to  determine  the  value  and  usability  of  several  computer/user 
interfaces  for  tasks  that  require  browsing  or  retrieving  information.  We  will  not  be 
evaluating  you,  but  how  well  the  interfaces  help  you  perform  a  task.  Once  you  begin  the 
study,  it  will  take  approximately  from  three  and  a  half  to  four  hours  of  your  time. 

PROCEDURE 

Before  the  experiment  starts,  you  will  be  asked  to  fill  out  a  questionnaire  concerning 
demographic  and  educational  factors  as  well  as  experience  and  attitudes  that  might  affect 
your  performance.  You  may  also  be  asked  to  take  a  short  test  of  your  visual  perception. 

In  the  experiment,  you  will  be  seated  in  front  of  a  computer.  You  will  first  work  through  a 
tutorial  on  the  type  of  problem  you  will  be  solving  and  on  the  interface  you  will  be  using. 
You  will  then  be  presented  with  text  describing  an  information  request.  After  you  have  read 
the  description,  you  will  be  asked  to  browse  an  on-line  collection  of  documents  to  find 
documents  relevant  to  as  many  “aspects”  of  the  information  request  as  you  can  in  20 
minutes.  (“Aspects”  will  be  explained  in  the  tutorial.)  At  the  end  of  20  minutes,  you  will  be 
asked  to  start  another  problem,  and  at  the  end  of  the  second  20-minute  period,  you  will  be 
asked  to  start  a  third  problem.  After  you  have  finished  the  third  problem,  you  will  be 
trained  on  a  new  interface  and  will  then  try  three  additional  problems.  Thus,  you  will  work 
on  six  problems  overall. 

Your  use  of  the  computer  will  require  reading  text  and  graphics  on  the  computer  monitor, 
as  well  as  using  the  keyboard  and  the  mouse  to  manipulate  the  display.  The  computer  will 
record  your  judgements  on  documents  as  well  as  any  changes  in  those  judgements.  The 
experiment  will  also  be  videotaped. 

After  you  have  completed  the  experiment,  you  will  participate  in  a  short  debriefing  session. 

RISKS  AND  BENEFITS 

If  you  complete  the  study,  you  will  be  compensated  $35.  If  you  withdraw  from  the  study 
for  any  reason,  you  will  not  be  compensated. 

This  study  involves  no  risk  on  your  part  and  you  will  receive  no  direct  benefit  other  than 
the  cash  compensation. 

CONFIDENTIAUITY 

The  information  which  is  recorded  in  the  computer  during  the  running  of  the  experiment 
will  not  contain  any  indication  of  your  name  or  other  personal  data.  In  addition,  data  we 
collect  from  you  during  this  study  will  not  be  associated  with  your  name  in  any  analysis  or 
report. 
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Your  social  security  number  will  be  needed  for  processing  the  cash  compensation  but  will 
not  be  used  for  any  other  purpose  and  will  not  be  kept  except  in  records  required  for 
accounting  purposes.  It  will  not  be  reported  anywhere  other  than  the  accounting  office. 

The  videotape  of  the  experiment  will  be  used  solely  to  help  in  evaluating  and  improving  the 
system;  it  will  not  be  distributed  or  shown  to  anyone  not  associated  with  the  evaluation 
process. 

SAFEGUARDS 

At  any  time  during  the  experiment,  you  are  free  to  leave  for  whatever  reason.  Your 
participation  in  this  study  is  purely  voluntary.  Additionally,  you  are  under  no  requirement 
to  complete  all  sessions.  If  you  experience  any  discomfort,  you  should  report  it 
immediately  to  the  experimenter  and/or  to  the  Human  Subject  Review  Committee  (see 
below  for  phone  and  address). 

If  you  are  a  student,  whether  or  not  you  complete  this  study  will  not  affect  your  course 
work  or  grade  in  any  class. 

OTHER  INFORMATION 

We  would  appreciate  it  if  you  would  not  discuss  this  session  with  anyone  who  you  know 
or  believe  will  be  participating  but  who  has  not  yet  done  so. 

After  all  participants  have  been  run  in  the  experiment,  the  data  will  be  summarized.  At  that 
point,  you  may  decide  you  would  like  to  see  how  individuals  performed  overall;  if  so,  a 
copy  of  the  summary  data  can  easily  be  made  available  to  you.  You  should  contact  the 
experimenter  approximately  6-8  weeks  after  you  have  run  in  the  study  should  you  desire 
more  information. 


After  reading  the  above  form,  please  sign  below  if  you  choose  to  continue  in  the 
experiment.  Your  signature  simply  indicates  that  you  have  read  and  understood  the  above 
information  and  that  you  wish  to  continue  in  the  experiment. 

Signature: _ 

Print  Name: _ 


Experimenter: 

Dr.  James  Allan 

Department  of  Computer  Science 
Lederle  Graduate  Research  Center 
University  of  Massachusetts 
Amherst,  MA  01003 
413/545-0463 


Additional  Point  of  Contact: 

Human  Subjects  Review  Committee 

Room  512 

Goodell  Building 

University  of  Massachusetts 

Amherst,  MA  01003 

413/545-0668 


22 


UMASS  TREC-6  INTERACTIVE  SEARCHING  STUDY 
ENTRY  (PRE-SEARCH)  QUESTIONNAIRE 

Searcher  _ 

Condition 

Please  list  all  the  college/university  degrees  that  you  have  (or  expect  to  have): 


Degree 

Major 

Date 

Degree 

Major 

Date 

Degree 

Major 

Date 

Degree 

Major 

Date 

What  is  your  age? 

□  Under  21  □  31-40  □  51-60 

□  21-30  □  41-50  □  Over  60 

What  is  your  gender? 

□  Female 

□  Male 

Have  you  participated  in  previous  TREC  Searching  Studies? 

□  Yes 

□  No 

Overall,  for  how  many  years  have  you  been  doing  on-line  searching?  years 
Who  have  you  performed  searches  for? 

□  Yourself 

□  Others 

□  Others  and  yourself 
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Have  you  ever  had  any  experience  using  any  of  the  following  information  retrieval 
systems? 


□  ZPRISE 

□  INQUERY 


P 


lease  circle  the  appropriate  number... 


How  much  experience  have  you  had... 

None 

Some 

A  lot 

using  computers 

1 

2 

3 

4 

5 

searching  on  computerized  library  catalogs 

1 

2 

3 

4 

5 

searching  on  CD-ROM  systems,  e.g.  Infotrac, 
Grolier 

1 

2 

3 

4 

5 

searching  on  commercial  on-line  systems,  e.g., 
Dialog,  Lexis,  BRS  Afterdark 

1 

2 

3 

4 

5 

searching  on  World  Wide  Web,  e.g.  Lycos,  Alta 
Vista,  Infoseek,  Excite 

1 

2 

3 

4 

5 

searching  on  other  systems  (please  specify  the 
system): 

1 

2 

3 

4 

5 

searching  full-text  databases 

1 

2 

3 

4 

5 

searching  in  ranked-output  information  retrieval 
systems 

1 

2 

3 

4 

5 

searching  in  information  retrieval  systems  that 
provide  automatic  relevance  feedback 

1 

2 

3 

4 

5 

using  a  mouse-based  interface 

1 

2 

3 

4 

5 

using  3D  interfaces  (e.g.  3D  on-line  chat  rooms,  3D 
first-person  action  games) 

1 

2 

3 

4 

5 
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UMASS  TREC-6  INTERACTIVE  SEARCHING  STUDY 
SEARCHER  INTERVIEW 


Searcher  _ 

Condition  _ 

In  order  to  understand  the  different  searching  experiences  of  our  participants,  we  would 
like  you  to  answer  a  couple  of  questions  about  the  methods  that  you  typically  use  when  you 
do  on-line  searching.  When  you  answer  these  questions,  please  try  to  give  us  as  much 
detail  as  you  can.  Please  use  this  worksheet  for  any  notes  that  you  wish  to  make. 

Imagine  that  you  are  interested  in  learning  about  the  different  alternative  sources  of  energy 
for  automobiles.  You  decide  to  investigate  the  literature  by  using  a  computerized  database 
of  newspaper  articles  that  is  available  for  your  use.  Since  you  are  interested  in  identifying 
as  many  "aspects"  of  this  topic  as  possible,  you  want  to  identify  each  one  of  the  different 
alternative  sources  of  energy  for  automobiles,  including  gasoline  additives  that  decrease 
pollution  or  reduce  oil  consumption. 

1.  What  steps  would  you  take  in  order  to  perform  a  search  that  would  identify  as  many 
"aspects"  as  possible  for  this  topic?  Describe  your  overall  approach  by  listing  what 
would  you  do  first,  and  then  describe  each  step  that  you  would  follow  after  that. 


2.  How  would  you  decide  that  your  search  is  finished? 
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UMASS  TREC-6  INTERACTIVE  SEARCHING  STUDY 
SEARCH  WORKSHEET 


Topic  Number _  Searcher _ 

Condition  _ 

Please  use  this  sheet  of  paper  to  write  down  any  notes  that  you  would  like  to  make  during 
your  search  and  to  list  the  aspects  as  you  identify  them  for  this  topic. 
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UMASS  TREC-6  INTERACTIVE  SEARCHING  STUDY 
POST-SEARCH  QUESTIONNAIRE 


Topic  Number _  Searcher _ 

Condition  _ 

Please  answer  the  following  questions,  as  they  relate  to  this  specific  topic. 

Please  circle  the  appropriate  number... 


To  what  extent... 

Not  at  all 

Somewhat 

Extremely 

are  you  familiar  with  this  topic? 

1  2 

3 

4 

5 

do  you  think  this  text  collection  is  suited 
to  find  the  answer  for  this  topic? 

1  2 

3 

4 

5 

was  it  difficult  to  do  this  search? 

1  2 

3 

4 

5 

are  you  satisfied  with  your  search  results? 

1  2 

3 

4 

5 

are  you  confident  that  you  identified  all 
the  possible  aspects  for  this  topic? 

1  2 

3 

4 

5 

are  you  satisfied  with  the  aspects  you 
have  retrieved  for  this  topic? 

1  2 

3 

4 

5 

are  you  satisfied  with  the  amount  of 
relevant  material  (versus  nonrelevant) 
retrieved  by  the  system? 

1  2 

3 

4 

5 

did  you  have  enough  time  to  do  an 
effective  search? 

1  2 

3 

4 

5 
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UMASS  TREC-6  INTERACTIVE  SEARCHING  STUDY 
EXIT  QUESTIONNAIRE 


Searcher  _ 

So  we  can  have  a  better  understanding  of  your  overall  searching  experience,  we  would  like 
to  ask  you  some  final  questions  about  your  experience  today.  Please  use  the  scale 
provided.  In  this  scale  a  "1"  means  "not  at  all",  a  "3"  means  "somewhat",  and  a  "5"  means 
"extremely". 

Please  consider  the  searching  experience  that  you  had  with  INQUERY's  aspect  window. 


Please  circle  the  appropriate  number... 


Rate  the  following... 

Not  at  all 

Somewhat 

Extremely 

How  useful  is  the  aspect  window? 

1 

2 

3 

4 

5 

How  easy  was  it  to  use  the  aspect 
window? 

1 

2 

3 

4 

5 

How  easy  was  it  to  learn  to  use  the 
aspect  window? 

1 

2 

3 

4 

5 

How  well  did  you  understand  how  to 
use  the  aspect  window? 

1 

2 

3 

4 

5 

Please  write  down  any  other  comments  that  you  have  about  the  aspect  window. 
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Please  consider  the  searching  experience  that  you  had  with  INQUERY's  3D  visualization. 


Please  circle  the  appropriate  number... 


Rate  the  following... 

Not  at  all 

Somewhat 

Extremely 

How  useful  is  the  3D  visualization? 

1 

2 

3 

4 

5 

How  easy  was  it  to  use  the  3D 
visualization? 

1 

2 

3 

4 

5 

How  easy  was  it  to  learn  to  use  the  3D 
visualization? 

1 

2 

3 

4 

5 

How  well  did  you  understand  how  to 
use  the  3D  visualization? 

1 

2 

3 

4 

5 

Please  write  down  any  other  comments  that  you  have  about  the  3D  visualization. 
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Please  consider  the  searching  experience  that  you  had  with  INQUERY. 


Please  circle  the  appropriate  number... 


Rate  the  following... 

Not  at  all 

Somewhat 

Extremely 

How  easy  was  it  to  use  this  information 
system? 

1 

2 

3 

4  5 

How  easy  was  it  to  learn  to  use  this 
information  system? 

1 

2 

3 

4  5 

How  well  did  you  understand  how  to 
use  this  information  system? 

1 

2 

3 

4  5 

Please  write  down  any  other  comments  that  you  have  about  your  searching  experience  with 
INQUERY  here.  Thank  you! 
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Please  consider  the  searching  experience  that  you  had  with  ZPRISE. 


Please  circle  the  appropriate  number... 


Rate  the  following... 

Not  at  all 

Somewhat 

Extremely 

How  easy  was  it  to  use  this  information 
system? 

1 

2 

3 

4 

5 

How  easy  was  it  to  learn  to  use  this 
information  system? 

1 

2 

3 

4 

5 

How  well  did  you  understand  how  to 
use  this  information  system? 

1 

2 

3 

4 

5 

Please  write  down  any  other  comments  that  you  have  about  your  searching  experience  with 
ZPRISE  here.  Thank  you! 
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Please  consider  the  entire  experience  that  you  just  had. 


Please  circle  the  appropriate  number... 


Rate  the  following... 

Not  at  all 

Somewhat 

Extremely 

To  what  extent  were  you  able  to 
understand  the  nature  of  the  task? 

1 

2 

3 

4 

5 

To  what  extent  did  you  find  this  task 
similar  to  other  searching  tasks  that  you 
typically  perform? 

1 

2 

3 

4 

5 

How  different  did  you  find  INQUERY 
from  ZPRISE? 

1 

2 

3 

4 

5 

Please  rank  the  two  systems  in  order  of  how  easy  they  were  to  use.  Place  a  "  1 "  next  to 
the  easiest  system  to  use  and  a  "2"  next  to  the  most  difficult  system  to  use. 

ZPRISE _ 

INQUERY _ 

Please  rank  the  two  systems  in  order  of  how  easy  they  were  to  learn  to  use.  Place  a 
"1"  next  to  the  easiest  system  to  leam  and  a  "2"  next  to  the  most  difficult  system  to  leam  to 
use. 


ZPRISE _ 

INQUERY _ 

Please  rank  the  two  systems  in  order  of  which  system  you  liked  the  best.  Place  a 
"1"  next  to  the  system  that  you  liked  the  best  and  a  "2"  next  to  the  system  you  liked  the 
least. 


ZPRISE _ 

INQUERY _ 

Please  write  down  any  other  comments  that  you  have  about  your  overall  searching 
experience  here.  Fell  free  to  use  the  other  page.  Thank  you ! 


32 


References 


[1]  Allan,  James,  Leouski,  Anton,  and  Swan,  Russell.  “Interactive  Cluster  Visualization  for 
Information  Retrieval”,  submitted  for  conference  publication. 

[2]  Chalmers,  M.  and  Chitson,  P.  (1992).  “Bead:  Explorations  in  Information  Visualization”, 
in  Proceedings  of  the  15th  International  ACM/SIGIR  Conference,  Copenhagen,  Denmark,  pp. 
330-337. 

[3]  Croft,  W.  B.  (1979).  Organizing  and  Searching  Large  Document  Collections,  Ph.D. 
Thesis,  University  of  Cambridge. 

[4]  Ekstrom,  R.B.,  French,  J.  W.,  Harman,  H.  H.,  and  Dermen,  D.  (1976).  Manual  for  Kit  of 
Factor-Referenced  Cognitive  Tests  1976.  Educational  Testing  Service,  Princeton,  New  Jersey. 

[5]  Harman,  Donna  (1997).  The  Fifth  Text  REtrieval  Conference  (forthcoming). 

[6]  Hearst,  Marti  A.  (1995).  “Visualization  of  Term  Distribution  Information  in  Full  Text 
Information  Access”,  in  CHI  ’95  Conference  Proceedings,  ACM  Press,  pp.  59-66. 

[7]  Hearst,  Marti  A.  and  Pedersen,  Jan  O.  (1996).  “Reexamining  the  Cluster  Hypothesis: 
Scatter/Gather  on  Retrieval  Results”,  in  Proceedings  of  SIGIR  '96  (Zurich,  Switzerland)  ACM 
Press. 

[8]  Swan,  Russell  C  and  Allan,  James  (1996).  “Improving  Interactive  Information  Retrieval 
Effectiveness  with  3-D  Graphics”.  Tech  Report  TR-96,  IR-100,  University  of  Massachusetts 
Computer  Science  Department. 

[9]  TREC  Interactive  Specification  (1997),  at  http://www-nlpir.nist.gov/~over/t6i  as  of  late 
August  1997. 

[10]  van  Rijsbergen,  C.  J  (1979).  Information  Retrieval,  second  edition,  Butterworths. 

[11]  Veerasamy,  Aravindan  and  Belkin,  Nicholas  J.  (1996).  “Evaluation  of  a  Tool  for 
Visualization  of  Information  Retrieval  Results”,  in  Proceedings  of  SIGIR  '96,  pp.  85-92. 


33 


